{
  "cells": [
    {
      "metadata": {
        "_uuid": "0f0857b69f9baca8c614089b437fcf61a09c659c"
      },
      "cell_type": "markdown",
      "source": "# A quick and simple GB model optimisation on EXT\\_SOURCE\\_\\* variables\nThis kernel has started from the simple and clear [15 lines: Just EXT_SOURCE_x](https://www.kaggle.com/lemonkoala/15-lines-just-ext-source-x) by [Lem Lordje Ko](https://www.kaggle.com/lemonkoala). Goal goal is to see what performance can one reach in short piece of code. What has been added on top on the original kernel is optimisation of LightGBM hyper-parameters. The final reported precision is 0.723 locally and 0.712 on the public leaderboard"
    },
    {
      "metadata": {
        "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5",
        "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
        "trusted": true,
        "collapsed": true
      },
      "cell_type": "code",
      "source": "import pandas as pd \nimport numpy as np\nimport lightgbm as lgb\n\ndata = pd.read_csv(\"../input/application_train.csv\")\ntest = pd.read_csv(\"../input/application_test.csv\")",
      "execution_count": 2,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "a46281b8cbf3a8e70a87a18bf090d7905ee350d9"
      },
      "cell_type": "markdown",
      "source": "Define parameter range in which optimisation will be performed."
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": true,
        "_uuid": "0f62f8e4c12280d473cc419dcccfcb50fa128f93"
      },
      "cell_type": "code",
      "source": "from scipy.stats import randint as sp_randint\nfrom scipy.stats import uniform as sp_uniform\nparam_test ={'num_leaves': sp_randint(6, 50), \n             'min_child_weight': sp_randint(1, 500), \n             'colsample_bytree': sp_uniform(loc=0.6, scale=0.4), \n             'subsample': sp_uniform(loc=0.2, scale=0.8), \n             'reg_alpha': [0, 1e-1, 1, 2, 5, 7, 10, 50, 100],\n             'reg_lambda': [0, 1e-1, 1, 5, 10, 20, 50, 100]}\n",
      "execution_count": 48,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "68a383bf9c0968759bfb19316f69c22445965d50"
      },
      "cell_type": "markdown",
      "source": "Define the hyper-parameter optimiser, it will test `n_HP_points_to_test` points sampled randomly. Beware: 3x20 (`CV_folds x n_HP_points_to_test`)  will run for approx 3 min on 4 CPU cores on kaggle"
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": true,
        "_uuid": "54af8323f33a4e000480f78d12a6bc649f99819b"
      },
      "cell_type": "code",
      "source": "n_HP_points_to_test = 20\nfrom sklearn.model_selection import RandomizedSearchCV\nclf = lgb.LGBMClassifier(max_depth=-1, is_unbalance=True, random_state=314, silent=True, metric='None', n_jobs=5)\ngs = RandomizedSearchCV(\n    estimator=clf, param_distributions=param_test, \n    n_iter=n_HP_points_to_test,\n    scoring='roc_auc',\n    cv=5,\n    refit=True,\n    random_state=314,\n    verbose=True)",
      "execution_count": 49,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "5183fb4b294cfd6bd6f18c6f1b46912158ade91f"
      },
      "cell_type": "markdown",
      "source": "Do actual parameter tune"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "bb6a3bc1244f911d50ca01d157e785bb46fbae39",
        "collapsed": true
      },
      "cell_type": "code",
      "source": "gs.fit(data.filter(regex=r'^EXT_SOURCE_.', axis=1), data['TARGET'])\nprint('Best score reached: {} with params: {} '.format(gs.best_score_, gs.best_params_))",
      "execution_count": 50,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "b1b978463332fb27f885bf251776e4627609a60c"
      },
      "cell_type": "markdown",
      "source": "Let's print the 5 best parameter sets based on the average roc auc on the testing fold in CV"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "614e1bd9251b9944dadbf30ac69ba0a682beb913",
        "collapsed": true
      },
      "cell_type": "code",
      "source": "print(\"Valid+-Std     Train  :   Parameters\")\nfor i in np.argsort(gs.cv_results_['mean_test_score'])[-5:]:\n    print('{1:.4f}+-{3:.4f} {2:.4f}   :  {0}'.format(gs.cv_results_['params'][i], \n                                    gs.cv_results_['mean_test_score'][i], \n                                    gs.cv_results_['mean_train_score'][i],\n                                    gs.cv_results_['std_test_score'][i]))",
      "execution_count": 57,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "11c9add781cc5b3617b60201d45e65d65c8ed1bb"
      },
      "cell_type": "markdown",
      "source": "Prepare a submission (note that you can directly submit it from the `Output` tab of the kernel, when you fork it)"
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": true,
        "_uuid": "8fed4a77dc999d917d1bf27fe39fc241985939a3"
      },
      "cell_type": "code",
      "source": "probabilities = gs.best_estimator_.predict_proba(test.filter(regex=r'^EXT_SOURCE_.', axis=1))\nsubmission = pd.DataFrame({\n    'SK_ID_CURR': test['SK_ID_CURR'],\n    'TARGET':     [ row[1] for row in probabilities]\n})\nsubmission.to_csv(\"submission.csv\", index=False)",
      "execution_count": 19,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "collapsed": true,
        "_uuid": "cbcb412366346deeedab895e197861f5db9a82df"
      },
      "cell_type": "code",
      "source": "",
      "execution_count": null,
      "outputs": []
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "version": "3.6.5",
      "mimetype": "text/x-python",
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "pygments_lexer": "ipython3",
      "nbconvert_exporter": "python",
      "file_extension": ".py"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 1
}